Skip to content

[SLES-2971] feat(traces): gate X-Ray-driven sampling behind DD_MERGE_XRAY_TRACES - #1337

Closed
lym953 wants to merge 1 commit into
yiming.luo/sles-2971-xray-sampled-priorityfrom
yiming.luo/sles-2971-merge-xray-gate
Closed

[SLES-2971] feat(traces): gate X-Ray-driven sampling behind DD_MERGE_XRAY_TRACES#1337
lym953 wants to merge 1 commit into
yiming.luo/sles-2971-xray-sampled-priorityfrom
yiming.luo/sles-2971-merge-xray-gate

Conversation

@lym953

@lym953 lym953 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Stacked on #1325 — review that first; this diff is only the gate.

Problem

After #1325, an AWS-generated Sampled=1 still becomes an explicit Datadog priority of 1. X-Ray's sampling decision therefore still drives Datadog's, and a user who wants their own sampling to govern has no way to decline.

The tracer libraries make this opt-in through DD_MERGE_XRAY_TRACES (DD_MERGE_DATADOG_XRAY_TRACES in Ruby), which defaults to false. bottlecap has never read it.

Background

For an SQS trigger, the extension falls back to the X-Ray AWSTraceHeader when the record carries no _datadog attribute. Any Lambda→SQS→Lambda chain has such a header, because the AWS SDKs plant one for recursion detection, so this fallback fires for pipelines that have nothing to do with X-Ray.

Change

Adds a merge_xray_traces config (DD_MERGE_XRAY_TRACES, default false) and gates the flag on it:

let honor_sampled_flag = datadog_planted || merge_xray_traces;
Header found on the message Priority sent, merge off (default) Priority sent, merge on
Datadog-planted, Sampled=1 1 — keep 1 — keep
Datadog-planted, Sampled=0 0 — drop 0 — drop
AWS-generated, Sampled=1 1 — keep 1 — keep
AWS-generated, Sampled=0 none — the tracer decides 0 — drop

"Datadog-planted" means the root id carries the …-00000000<16 hex> padding a Datadog library writes; anything else is AWS-generated. Only the last row moves with the flag.

Default false, so behavior holds at #1325's until a user opts in.

Datadog-planted headers stay ungated

dd-trace-java plants context in this same header for SQS, and it is the only carrier for JMS→SQS. Gating that on an X-Ray flag would break Datadog-to-Datadog propagation for users with no X-Ray involvement.

Correlation stays ungated

Trace and parent ids come from an AWS-generated header in both states. That is how bottlecap has always behaved, it costs no spans, and gating it would leave the reporting customer with their spans restored but their producer→consumer link gone.

Turning merging on reinstates the drop

With merging on, an AWS-generated Sampled=0 drops the trace, which is the reported behavior — restored deliberately for anyone who asks for it, since the flag means X-Ray's decisions count.

Where the flag is read

Off the propagator, which already carries the config to every extraction site. Threading the config explicitly would touch five more files, including two request-state tuples.

Naming

This reuses DD_MERGE_XRAY_TRACES rather than introducing a new variable, so users find the setting where they already expect it. That broadens what the name covers: the docs currently scope trace merging to Node.js and Python and describe it as merging X-Ray's own spans, so they need a line for this path.

Testing

Verified live on the SLES-2971 repro (Go, arm64, provided.al2023, SQS trigger), sending each header shape in both flag states. Every span in the trace carried the priority shown.

Header sent on the message Priority observed, merge off Priority observed, merge on
AWS-generated, Sampled=0 1 — kept 0 — dropped
AWS-generated, Sampled=1 1 — kept 1 — kept
Datadog-planted, Sampled=0 0 — dropped 0 — dropped
Datadog-planted, Sampled=1 1 — kept 1 — kept

Only the AWS-generated Sampled=0 row moves with the flag, which is the intent. A trace id was present in all eight runs, so correlation is unaffected either way.

🤖 Generated with Claude Code

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 21, 2026

Copy link
Copy Markdown

Pipelines

Unblock PR with BitsAI

⚠️ Warnings

🚦 4 Pipeline jobs failed

DataDog/datadog-lambda-extension | e2e-test-status (amd64) — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/datadog-lambda-extension | integration-suite: [payload-size] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

Bottlecap (Rust) | Audit

View more details · View in GitHub Actions

Critical vulnerabilities detected during Azure audit check. Audit failed due to found vulnerabilities and maintenance issues.

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 006b1f7 | Docs | View more details | Give us feedback!

…XRAY_TRACES

Adopting an AWS-generated X-Ray `Sampled` flag as a Datadog sampling decision is
opt-in behavior in the tracer libraries, where `DD_MERGE_XRAY_TRACES` defaults to
false. bottlecap never read it, so there was no way to decline.

Add the config and gate the flag on it. A header a Datadog library planted is
never gated -- its `Sampled` is a Datadog decision regardless of the setting, and
gating it would break dd-trace-java's SQS propagation, which uses this same
header and has nothing to do with X-Ray.

Scope is the sampling verdict only. Trace and parent IDs are still taken from an
AWS-generated header either way, so correlation is unchanged by this flag; that
is pre-existing v88+ behavior and gating it is a separate argument.

The flag is read off `DatadogCompositePropagator`, which already carries the
config to every extraction site -- threading `Arc<Config>` through the listener,
LWA proxy, and interceptor state instead would have touched five more files.
@lym953
lym953 force-pushed the yiming.luo/sles-2971-merge-xray-gate branch from 9e2f4db to 006b1f7 Compare August 25, 2026 03:13
@lym953

lym953 commented Aug 25, 2026

Copy link
Copy Markdown
Contributor Author

Closing: the gate isn't the right shape. DD_MERGE_XRAY_TRACES governs whether X-Ray's spans join a Datadog trace, not whether X-Ray's sampler decides Datadog's sampling — so an X-Ray drop should not propagate whether merging is on or off, and there is nothing for a config flag to gate. #1325 already implements that: a Datadog-planted header's Sampled is honored, an AWS-generated one never produces a drop.

Branch yiming.luo/sles-2971-merge-xray-gate is left in place if the correlation-gating question is picked up later.

@lym953 lym953 closed this Aug 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant